Add support for fetching Redshift query results using Redshift unload command #24117

mayankvadariya · 2024-11-13T00:50:29Z

Description

Redshift supports unloading select query results to S3 using UNLOAD command. PR aims to use unload command to unload Redshift query results on S3 and to read generated redshift query results files in parallel.

Official documentation on Redshift unload command:
https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html
https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD_command_examples.html

Connector internally converts Trino SQL to Redshift's unload SQL and runs it. Resultant Parquet files containing query results will be read in parallel to generate Trino query result. Only Parquet file format is supported in Redshift unload command.

Below are certain limitations for which Trino internally force fallback to traditional JDBC even if connector is configured to use unload.

select query containing TimeType and VarbinaryType column type is unsupported as unload doesn't support unloading to Parquet format.
select query ending with limit clause is unsupported as unload command doesn't support limit clause.
select query containing filter condition on DecimalType column is unsupported with unload as unload query is generated by Trino doesn't add precision to numeric cast function.
select query projecting static literals.

Benchmarking

Setup information:

Table	Schema	Records	Data size(agg. parquet file size)
nation	tpch.sf1000	25	3.5K
supplier	tpch.sf1000	10000000	1.4G
customer	tpch.sf1000	150000000	22.8G

Coordinator size: m6g.2xlarge
Worker size: 8 X m7g.8xlarge

Redshift cluster size: 2(nodes) X dc2.large

Benchmarking stats

UNLOAD performs better as the volume of query results increases .

Query	Query text	Average JDBC flow execution time(ms)	Average UNLOAD flow execution time(ms)
q01	SELECT checksum(custkey), checksum(name), checksum(address), checksum(nationkey), checksum(phone), checksum(acctbal), checksum(mktsegment), checksum(comment) FROM "customer";	622,571	240,893
q02	SELECT checksum(suppkey), checksum(name), checksum(address), checksum(nationkey), checksum(phone), checksum(acctbal), checksum(comment) FROM "supplier";	35,618	17,327
q03	SELECT checksum(nationkey), checksum(name), checksum(regionkey), checksum(comment) FROM "nation";	331	398

I'd like to thank @raunaqmorarka for his constant support in helping me land this PR.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Redshift
* Improve performance of reading from Redshift tables using `UNLOAD` command on Redshift. ({issue}`24117`)

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitManager.java

docs/src/main/sphinx/connector/redshift.md

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftPageSource.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftSessionProperties.java

plugin/trino-redshift/src/test/java/io/trino/plugin/redshift/TestRedshiftUnload.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitManager.java

docs/src/main/sphinx/connector/redshift.md

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftConnectorFactory.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftPageSource.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitManager.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadConnector.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftPageSource.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftPageSourceProvider.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitManager.java

.github/workflows/ci.yml

mayankvadariya · 2024-12-23T22:06:48Z

https://github.com/trinodb/trino/actions/runs/12462869117/job/34784444202?pr=24359 Redshift tests are passing.
Additionally, added benchmarking stats in PR desc

core/trino-main/src/main/java/io/trino/operator/SplitOperatorInfo.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftClientModule.java

docs/src/main/sphinx/connector/redshift.md

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java

plugin/trino-redshift/src/test/java/io/trino/plugin/redshift/TestRedshiftConnectorTest.java

plugin/trino-redshift/src/test/java/io/trino/plugin/redshift/TestRedshiftUnload.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftParquetPageSource.java

docs/src/main/sphinx/connector/redshift.md

mosabua · 2024-12-24T22:49:06Z

docs/src/main/sphinx/connector/redshift.md

@@ -64,6 +64,32 @@ documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/jdbc20-configura
 ```{include} jdbc-authentication.fragment
 ```

+### UNLOAD configuration
+
+This feature enables using Amazon S3 to efficiently transfer data out of Redshift


Please describe what this actually does for user .. seems like it is only a performance improvement .. right? And if so .. why is it called unload?

This feature uses UNLOAD(https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html) approach for performance improvements. Additionally, it requires additional configs to enable this feature. Please suggest text improvements.

So lets move that into the performance section and explain more there.

I assume we need to explain that they need a S3 account and whatever else .. and when this should and should not be used.. I am not aware of this info so I cant really reword appropriately without more details

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java

plugin/trino-redshift/src/test/java/io/trino/plugin/redshift/TestRedshiftUnload.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftParquetPageSource.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java

raunaqmorarka

lgtm % comments

core/trino-main/src/main/java/io/trino/operator/SplitOperatorInfo.java

plugin/trino-redshift/src/test/java/io/trino/plugin/redshift/TestRedshiftUnload.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftSplitManager.java

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java

Previously SplitOperatorInfo wasn't Mergeable and hence base OperatorInfo(OperatorStats#getMergeableInfoOrNull) was null.

...no-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadJdbcQueryEventListener.java

Co-authored-by: Mayank Vadariya <[email protected]>

cla-bot bot added the cla-signed label Nov 13, 2024

Praveen2112 reviewed Nov 13, 2024

View reviewed changes

mayankvadariya force-pushed the mayank/redshift-unload branch from 51fae23 to 4732ea6 Compare November 14, 2024 01:47

github-actions bot added the docs label Nov 14, 2024

mayankvadariya force-pushed the mayank/redshift-unload branch 4 times, most recently from f5be0a8 to 7b7d095 Compare November 18, 2024 22:00

mayankvadariya requested review from Praveen2112, wendigo, raunaqmorarka, ebyhr, hashhar, anusudarsan and SemionPar November 19, 2024 14:18

mayankvadariya marked this pull request as ready for review November 19, 2024 14:20

SemionPar reviewed Nov 19, 2024

View reviewed changes

docs/src/main/sphinx/connector/redshift.md Outdated Show resolved Hide resolved

ebyhr reviewed Nov 22, 2024

View reviewed changes

findinpath reviewed Nov 22, 2024

View reviewed changes

docs/src/main/sphinx/connector/redshift.md Outdated Show resolved Hide resolved

raunaqmorarka reviewed Nov 25, 2024

View reviewed changes

findinpath reviewed Nov 25, 2024

View reviewed changes

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitManager.java Outdated Show resolved Hide resolved

anusudarsan reviewed Nov 25, 2024

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

mayankvadariya force-pushed the mayank/redshift-unload branch 2 times, most recently from a4868fc to a52f29b Compare November 26, 2024 22:35

github-actions bot added hudi Hudi connector iceberg Iceberg connector delta-lake Delta Lake connector hive Hive connector labels Nov 26, 2024

mayankvadariya force-pushed the mayank/redshift-unload branch 2 times, most recently from 4be8ea2 to e33c9a4 Compare November 27, 2024 03:25

mayankvadariya requested review from raunaqmorarka and ebyhr December 23, 2024 22:06

raunaqmorarka reviewed Dec 24, 2024

View reviewed changes

mosabua reviewed Dec 24, 2024

View reviewed changes

docs/src/main/sphinx/connector/redshift.md Outdated Show resolved Hide resolved

mosabua reviewed Dec 24, 2024

View reviewed changes

raunaqmorarka reviewed Dec 26, 2024

View reviewed changes

mayankvadariya force-pushed the mayank/redshift-unload branch from c9a1b30 to d35c9b2 Compare December 27, 2024 04:51

raunaqmorarka reviewed Dec 27, 2024

View reviewed changes

core/trino-main/src/main/java/io/trino/operator/SplitOperatorInfo.java Outdated Show resolved Hide resolved

plugin/trino-redshift/src/test/java/io/trino/plugin/redshift/TestRedshiftUnload.java Outdated Show resolved Hide resolved

mayankvadariya force-pushed the mayank/redshift-unload branch from d35c9b2 to 93308f8 Compare December 27, 2024 06:53

mayankvadariya requested a review from raunaqmorarka December 27, 2024 06:56

raunaqmorarka approved these changes Dec 27, 2024

View reviewed changes

findinpath reviewed Dec 27, 2024

View reviewed changes

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java Outdated Show resolved Hide resolved

findinpath reviewed Dec 27, 2024

View reviewed changes

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftSplitManager.java Show resolved Hide resolved

findinpath reviewed Dec 27, 2024

View reviewed changes

plugin/trino-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadSplitSource.java Show resolved Hide resolved

mayankvadariya force-pushed the mayank/redshift-unload branch from 93308f8 to c27227d Compare December 27, 2024 17:08

mayankvadariya and others added 2 commits December 27, 2024 14:20

Correctly merge multiple splits info

f33c4bb

Previously SplitOperatorInfo wasn't Mergeable and hence base OperatorInfo(OperatorStats#getMergeableInfoOrNull) was null.

Prepare to implement a page source provider for Redshift

9712af9

mayankvadariya force-pushed the mayank/redshift-unload branch from c27227d to 9582983 Compare December 27, 2024 19:21

raunaqmorarka reviewed Dec 31, 2024

View reviewed changes

...no-redshift/src/main/java/io/trino/plugin/redshift/RedshiftUnloadJdbcQueryEventListener.java Outdated Show resolved Hide resolved

mayankvadariya force-pushed the mayank/redshift-unload branch from 9582983 to 613cada Compare December 31, 2024 14:43

Fetch Redshift query results unloaded to S3

f8fdfb1

Co-authored-by: Mayank Vadariya <[email protected]>

mayankvadariya force-pushed the mayank/redshift-unload branch 2 times, most recently from d0a5473 to f8fdfb1 Compare December 31, 2024 15:54

raunaqmorarka approved these changes Dec 31, 2024

View reviewed changes

raunaqmorarka added the performance label Dec 31, 2024

raunaqmorarka merged commit b382e03 into trinodb:master Dec 31, 2024
97 of 98 checks passed

github-actions bot added this to the 469 milestone Dec 31, 2024

This was referenced Jan 1, 2025

Add Trino 469 release notes #24509

Open

Improve docs for Redshift parallel read #24619

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for fetching Redshift query results using Redshift unload command #24117

Add support for fetching Redshift query results using Redshift unload command #24117

mayankvadariya commented Nov 13, 2024 •

edited

Loading

mayankvadariya commented Dec 23, 2024

mosabua Dec 24, 2024

mayankvadariya Dec 24, 2024 •

edited

Loading

mosabua Dec 27, 2024

mosabua Dec 27, 2024

raunaqmorarka left a comment

Add support for fetching Redshift query results using Redshift unload command #24117

Add support for fetching Redshift query results using Redshift unload command #24117

Conversation

mayankvadariya commented Nov 13, 2024 • edited Loading

Description

Benchmarking

Setup information:

Benchmarking stats

Additional context and related issues

Release notes

mayankvadariya commented Dec 23, 2024

mosabua Dec 24, 2024

Choose a reason for hiding this comment

mayankvadariya Dec 24, 2024 • edited Loading

Choose a reason for hiding this comment

mosabua Dec 27, 2024

Choose a reason for hiding this comment

mosabua Dec 27, 2024

Choose a reason for hiding this comment

raunaqmorarka left a comment

Choose a reason for hiding this comment

mayankvadariya commented Nov 13, 2024 •

edited

Loading

mayankvadariya Dec 24, 2024 •

edited

Loading